Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AG-1591, AG-1592: don't deploy unused containers, create DocumentDB cluster, allow connection from API container #17

Open
wants to merge 7 commits into
base: dev
Choose a base branch
from

Conversation

hallieswan
Copy link
Contributor

@hallieswan hallieswan commented Jan 16, 2025

Description

  • AG-1591: don't deploy unused containers (agora-mongo, agora-data, agora-api-docs)
  • IT-4241: create DocumentDB cluster
  • AG-1592: add secrets to connect to DocumentDB instance
  • Add missing agora-app environment variables
  • Update CODEOWNERS per Khai

Depends on Sage-Bionetworks/sage-monorepo#2967

Note: plan to add the bastion in a future PR, work tracked in IT-4243

Copy link

dpulls bot commented Jan 16, 2025

🎉 All dependencies have been resolved !

@hallieswan hallieswan changed the title AG-1591, AG-1592: don't deploy unused containers, add secrets to connect to existing DocumentDB instances AG-1591, AG-1592: don't deploy unused containers, create DocumentDB cluster, allow connection from API container Jan 22, 2025
src/docdb_stack.py Outdated Show resolved Hide resolved
src/docdb_stack.py Outdated Show resolved Hide resolved
src/docdb_stack.py Outdated Show resolved Hide resolved
src/docdb_stack.py Outdated Show resolved Hide resolved
app.py Show resolved Hide resolved
@hallieswan hallieswan marked this pull request as ready for review January 22, 2025 22:59
@hallieswan hallieswan requested review from a team as code owners January 22, 2025 22:59
@hallieswan hallieswan requested a review from zaro0508 January 22, 2025 22:59
src/docdb_props.py Show resolved Hide resolved
) -> None:
super().__init__(scope, construct_id, **kwargs)

self.master_password_secret = sm.Secret(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense! Will the password environment variable in the API container be automatically updated when the secret is rotated?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, i believe that's the best feature of secrets manager. here's a blog on how to setup https://aws.amazon.com/blogs/security/how-to-rotate-amazon-documentdb-and-amazon-redshift-credentials-in-aws-secrets-manager/

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the article you linked, the application retrieves the password directly from Secrets Manager at runtime. However, the Agora API container currently retrieves the password from an environment variable, which is set by the ServiceStack during deployment. If the secret is retrieved at deployment time, then would the API container environment variable only be updated when the stack is redeployed (and not automatically be updated when the secret is rotated)?

If that's right, then to allow us to use automatic password rotation, I think we'd need to update the API to retrieve the password directly from Secrets Manager (and refetch the credentials without restarting the container, as described here) at runtime, rather than getting the password from an environment variable. What do you think?

Copy link
Contributor

@zaro0508 zaro0508 Jan 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ahh, yes you are correct. getting the secret from an env var will not work with secrets rotation. refactoring the API to get secrets directly from the secrets manager would be a better approach because is more secure and works with secrets rotation. I guess we can leave out secrets rotation at this time. I'll let you guys decide whether you want to refactor Agora to support this use case. I would highly recommend it though.

family="docdb5.0",
parameters={
"audit_logs": "disabled",
"audit_logs": "disabled",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

duplicate parameter?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will remove in the next commit


self.master_password_secret = sm.Secret(
self,
f"{construct_id}-master-password",
Copy link
Contributor

@zaro0508 zaro0508 Jan 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the convention used in other files for construct_id is camel case. Also i don't think you don't need to get fancy here, you can just set to something like "DocDbMasterPassword"

"tls": "disabled",
"ttl_monitor": "disabled",
},
db_cluster_parameter_group_name=f"{construct_id}-cluster-parameter-group",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

name parameter does not seem like it's required? if a name parameter is not required I would leave it out and let AWS name the resource. You can always dynamically get a resource name from its Ref.

)
self.docdb_security_group.add_ingress_rule(
peer=self.access_docdb_security_group,
connection=ec2.Port.tcp_range(27017, 27030),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

27017 is used multiple times in this file. would it make sense to create a variable for this?

Comment on lines +135 to +136
self.security_groups = props.container_security_groups.copy()
self.security_groups.append(self.security_group)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, it looks like we add the security_group defined here to every container so how about we move the security_group to the ServiceProps class? It can just be added to the container_security_groups object as a default SG?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we create the SecurityGroup in ServiceProps, then I believe ServiceProps would need to inherit from Construct, which would result in duplicate code between the props and the stack class. What do you think about continuing to create security_group in ServiceStack, so that the resource creation only happens in one class?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about continuing to create security_group in ServiceStack, so that the resource creation only happens in one class?

Not sure exactly what you mean but if you mean define a list of service groups in app.py then pass it to ServiceProps object then I think that will work as well.

@@ -171,6 +131,7 @@
"APP_VERSION": f"{agora_version}",
"CSR_API_URL": f"http://{fully_qualified_domain_name}/api/v1",
"SSR_API_URL": "http://agora-api:3333/v1",
"TAG_NAME": f"agora/v${agora_version}",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't this a dup of APP_VERSION?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They're very similar, but not quite the same (e.g. 4.0.0 for APP_VERSION vs agora/v4.0.0.0 for TAG_NAME). @sagely1 would know better, but I believe separate variables were created because the GitHub tag format was in flux. However, with that decision made and the format so similar, they should probably be collapsed to just one environment variable, but that will require work in the sage-monorepo first!

Copy link

@sagely1 sagely1 Jan 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have settled on semantic versioning after some discussion. TAG_NAME will be a string of a format like so:
agora/vX.X.X with a potential suffix such as -rc.X so agora/vX.X.X-rc.X. I think we can just change this to get a tag_name variable instead. https://semver.org/

Copy link
Contributor Author

@hallieswan hallieswan Jan 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great! I've opened a ticket to track the work, since we'll need to update the sage-monorepo code first.

app.py Outdated
Comment on lines 132 to 133
"CSR_API_URL": f"http://{fully_qualified_domain_name}/api/v1",
"SSR_API_URL": "http://agora-api:3333/v1",
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may need these to be https considering that the app will be served over https

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good, I can change that in the next commit

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants